Interpreting Audio/Video Recordings and Logs

Our experience using a combination of audio/video (A/V) recordings and logging to study long-term use of a computerized whiteboard illustrates that these techniques can provide insights into the use of applications. Logs captured information about the users’ actions in a form that supported automatic queries. A/V recordings captured information about the user’s goals and the system’s responses that informed our interpretation of the logs. Ultimately, we discovered that defects in the logs made it impossible to answer many of our questions. We were unable to use the A/V recordings as a substitute for the missing information in the logs; the cost of capturing and analyzing the data was too great. Based on our experience, we propose a set of guidelines for implementing logging that may lead to better results in future efforts to combine these techniques.